Current computer vision models, unlike the human visual system, cannot yet achieve general-purpose visual understanding. Existing efforts to create a general vision model are limited in the scope of assessed tasks and offer no overarching framework to perform them holistically. We present a new comprehensive benchmark, General-purpose Visual Understanding Evaluation (G-VUE), covering the full spectrum of visual cognitive abilities with four functional domains $\unicode{x2014}$ Perceive, Ground, Reason, and Act. The four domains are embodied in 11 carefully curated tasks, from 3D reconstruction to visual reasoning and manipulation. Along with the benchmark, we provide a general encoder-decoder framework to allow for the evaluation of arbitrary visual representation on all 11 tasks. We evaluate various pre-trained visual representations with our framework and observe that (1) Transformer-based visual backbone generally outperforms CNN-based backbone on G-VUE, (2) visual representations from vision-language pre-training are superior to those with vision-only pre-training across visual tasks. With G-VUE, we provide a holistic evaluation standard to motivate research toward building general-purpose visual systems via obtaining more general-purpose visual representations.

Modern virtual assistants use internal semantic parsing engines to convert user utterances to actionable commands. However, prior work has demonstrated that semantic parsing is a difficult multilingual transfer task with low transfer efficiency compared to other tasks. In global markets such as India and Latin America, this is a critical issue as switching between languages is prevalent for bilingual users. In this work we dramatically improve the zero-shot performance of a multilingual and codeswitched semantic parsing system using two stages of multilingual alignment. First, we show that constrastive alignment pretraining improves both English performance and transfer efficiency. We then introduce a constrained optimization approach for hyperparameter-free adversarial alignment during finetuning. Our Doubly Aligned Multilingual Parser (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively and outperforms XLM-R and mT5-Large using 3.2x fewer parameters.

对于许多下游任务（例如，情感分析，关系检测等），脑电图（EEG）和语言已被广泛探索。研究这两个领域的多模式方法尚未得到很好的探索，即使近年来，多模式学习被认为比单峰对应物更强大。在这项研究中，我们希望探索脑电图与语言之间的关系和依赖性，即一个领域如何反映和代表另一个领域。为了研究表示级别的关系，我们引入了MTAM（一种多模式变压器对准模型），以观察两种模式之间的协调表示，因此采用了转换表示来进行下游应用。我们使用各种关系对齐的寻求对准技术，例如规范相关性分析和Wasserstein距离，作为转化低级语言的损失函数，并将EEG特征转化为高级转化的特征。在下游应用程序，情感分析和关系检测上，我们在两个数据集（Zuco和k-emocon）上实现了新的最新结果。我们的方法在K-Emocon的情感分析中获得了16.5％的F1得分提高，对Zuco的情感分析的26.6％，以及对Zuco的关系检测的31.1％。此外，我们通过以下方式提供对性能改进的解释：（1）可视化原始特征分布和变换的特征分布，显示对齐模块发现和编码脑电图与语言之间的关系的有效性； （2）可视化单词级别和句子级的脑电图对齐权重，显示不同语言语义和脑电图频率特征的影响； （3）可视化大脑地形图，以提供有关大脑区域中脑电图和语言反应的连通性的直观演示。

本文提出了一种新的点云卷积结构，该结构学习了SE（3） - 等级功能。与现有的SE（3） - 等级网络相比，我们的设计轻巧，简单且灵活，可以合并到一般的点云学习网络中。我们通过为特征地图选择一个非常规域，在模型的复杂性和容量之间取得平衡。我们通过正确离散$ \ mathbb {r}^3 $来完全利用旋转对称性来进一步减少计算负载。此外，我们采用置换层从其商空间中恢复完整的SE（3）组。实验表明，我们的方法在各种任务中实现了可比或卓越的性能，同时消耗的内存和运行速度要比现有工作更快。所提出的方法可以在基于点云的各种实用应用中促进模棱两可的特征学习，并激发现实世界应用的Equivariant特征学习的未来发展。

语言规划旨在通过分解为更简单的低级步骤来实现复杂的高级目标。这种程序推理能力对于诸如家用机器人和虚拟助手等应用至关重要。尽管语言规划是日常生活中人类的基本技能，但对于缺乏现实世界中缺乏深层常识性知识的大型语言模型（LLM）来说，这仍然是一个挑战。以前的方法需要手动示例或带注释的程序才能从LLM中获取此类能力。相比之下，本文提出了神经符号的因果语言规划师（CLAP），该策划者通过注入常识的提示从LLM中引起了程序知识。 LLMS中的预训练知识本质上是一种未观察到的混杂因素，它在任务和行动计划之间引起虚假的相关性。通过结构性因果模型（SCM）的镜头，我们提出了一个有效的策略，以构建提示作为对SCM的因果干预。我们的策略使用图形采样技术和符号程序执行者，正式从常识知识基础上形成结构化因果提示。拍手在Wikihow和机器人上获得最新的表现，在反事实环境下，人类评估的相对提高了5.28％。这表明在语义和顺序的因果语言规划中拍手的优势。

密集的视频字幕旨在确定输入视频中感兴趣的事件，并为每个事件生成描述性标题。先前的方法通常遵循两个阶段的生成过程，该过程首先提出了每个事件的段，然后为每个已确定的细分市场提供标题。大规模序列产生预处理的最新进展在统一各种任务的任务制定方面取得了巨大的成功，但是到目前为止，更复杂的任务（例如密集的视频字幕）无法完全利用这种强大的范式。在这项工作中，我们展示了如何将密集视频字幕的两个子任务与一个序列生成任务建模，并同时预测事件和相应的描述。在YouCook2和Vitt上进行的实验表现出令人鼓舞的结果，并表明训练复杂任务的可行性，例如集成到大规模预处理模型中的端到端密集的视频字幕。

我们提出了Gan监督的学习，一个学习歧视模型的框架及其GAN生成的培训数据结束结束。我们将框架应用于密集的视觉调整问题。灵感来自经典的凝固方法，我们的甘蓝算法列举了空间变压器来将随机样本从受过协调的数据训练到常见的共同学习的目标模式。我们在八个数据集上显示结果，所有这些都证明了我们的方法成功对齐复杂数据并发现密集的对应。甘蓝显着优于过去自我监督的对应算法，并在几个数据集上与（有时超过）最先进的监督对应算法进行了近几个数据集 - 而不利用任何通信监督或数据增强，尽管仅仅是完全培训在GaN生成的数据上。对于精确的对应，我们通过最先进的受监管方法提高了3倍。我们展示了我们对下游GaN训练的图像数据集的增强现实，图像编辑和自动预处理的应用。

这些年来跨域情绪分类一直是一个热点，旨在使用来自源域的标记数据来学习可靠的分类器，并在目标域上进行评估。在此静脉中，大多数方法利用域适应，将数据从不同域映射到共同的特征空间。为了进一步提高模型性能，提出了针对挖掘域特定信息的几种方法。但是，其中大多数仅利用有限的域特定信息。在这项研究中，我们首先通过基于主题信息制定一种提取特定域的单词的方法。然后，我们提出了一个主题驱动的自适应网络（TDAN），用于跨域情绪分类。该网络由两个子网络组成：语义注意网络和域特定的单词注意网络，其结构基于变压器。这些子网采用不同的输入形式，并且它们的输出被融合为特征向量。实验验证了我们TDAN对跨域情绪分类的有效性。

We present Muse, a text-to-image Transformer model that achieves state-of-the-art image generation performance while being significantly more efficient than diffusion or autoregressive models. Muse is trained on a masked modeling task in discrete token space: given the text embedding extracted from a pre-trained large language model (LLM), Muse is trained to predict randomly masked image tokens. Compared to pixel-space diffusion models, such as Imagen and DALL-E 2, Muse is significantly more efficient due to the use of discrete tokens and requiring fewer sampling iterations; compared to autoregressive models, such as Parti, Muse is more efficient due to the use of parallel decoding. The use of a pre-trained LLM enables fine-grained language understanding, translating to high-fidelity image generation and the understanding of visual concepts such as objects, their spatial relationships, pose, cardinality etc. Our 900M parameter model achieves a new SOTA on CC3M, with an FID score of 6.06. The Muse 3B parameter model achieves an FID of 7.88 on zero-shot COCO evaluation, along with a CLIP score of 0.32. Muse also directly enables a number of image editing applications without the need to fine-tune or invert the model: inpainting, outpainting, and mask-free editing. More results are available at https://muse-model.github.io

We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for "scored actors" in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.